Skip to content

TST/CI: skipif numba tests on Ubuntu ARM for numba 0.61 #60847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Feb 7, 2025

Conversation

mroeschke
Copy link
Member

cc @phofl could use help looking into the failure in test_frame_setitem_dask_array_into_new_col

https://github.com/pandas-dev/pandas/actions/runs/13127760620/job/36627207210

@mroeschke mroeschke added the Testing pandas testing functions or related to the test suite label Feb 4, 2025
@mroeschke mroeschke added this to the 2.3 milestone Feb 4, 2025
@phofl
Copy link
Member

phofl commented Feb 4, 2025

I'll take a look

Did this start failing recently?

@mroeschke
Copy link
Member Author

Did this start failing recently?

Yeah started failing this week. Appears the Dask version is the same between the pass/failing runs, but maybe it's because the numpy version is different between them (passes with 2.0.2, fails with 2.1.3)

@phofl
Copy link
Member

phofl commented Feb 5, 2025

Thanks for the ping.

Looks like numpy changed something under the hood that exposed a bug in Dask...

dask/dask#11697 fixes this

@jorisvandenbossche
Copy link
Member

Looks like numpy changed something under the hood that exposed a bug in Dask...

dask/dask#11697 fixes this

Thanks @phofl!
Do I understand this correctly that our setitem code path with a dask array at some point goes through a np.array(arry_like, copy=False) call? (based on dask/dask#11697 (comment)) Because regardless of the fact that dask should align with upstream numpy and deprecate then raise if copy=False is not possible, this also seems something we should further investigate on the pandas side? (with CoW, I would think we shouldn't be setting an array-like with copy=False, but ensure we always set a copy of the values in the column)

@jorisvandenbossche
Copy link
Member

@mroeschke and thanks for the CI fixes! For the numba failures, did this also start to happen with a new numba release? Are they aware of the issue?

@jorisvandenbossche
Copy link
Member

And another segfault in pandas/tests/window/test_expanding.py

@phofl
Copy link
Member

phofl commented Feb 6, 2025

No, Dask was ignoring the copy kwarg and we were erroneously not copying by default if your array has a single chunk only. The copy kwarg of pandas was set to true

@mroeschke mroeschke changed the title TST: Apply skip/xfail markers for numba/dask updates TST/CI: skipif numba tests on Ubuntu ARM for numba 0.61 Feb 7, 2025
@mroeschke mroeschke added the numba numba-accelerated operations label Feb 7, 2025
@mroeschke
Copy link
Member Author

For the numba failures, did this also start to happen with a new numba release?

AFAICT, yes this happened when the Ubuntu ARM CI jobs started to install numba=0.61

Are they aware of the issue?

From a skim of the issue tracker, I don't think they are aware of the issue yet. But also, I haven't had a moment to come up with a MRE for the numba folks of the segfault

@mroeschke
Copy link
Member Author

Now that we're finally green here, going to merge to get the CI to green.

Opened #60879 to follow up on creating a MRE to report to numba

@mroeschke mroeschke merged commit 0a1577f into pandas-dev:main Feb 7, 2025
42 checks passed
@mroeschke mroeschke deleted the ci/tests branch February 7, 2025 23:32
Copy link

lumberbot-app bot commented Feb 7, 2025

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 2.3.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 0a1577f2f0d1c9efda4b7b2d616177691d1ca73b
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #60847: TST/CI: skipif numba tests on Ubuntu ARM for numba 0.61'
  1. Push to a named branch:
git push YOURFORK 2.3.x:auto-backport-of-pr-60847-on-2.3.x
  1. Create a PR against branch 2.3.x, I would have named this PR:

"Backport PR #60847 on branch 2.3.x (TST/CI: skipif numba tests on Ubuntu ARM for numba 0.61)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

mroeschke added a commit to mroeschke/pandas that referenced this pull request Feb 7, 2025
mroeschke added a commit that referenced this pull request Feb 8, 2025
…ntu ARM for numba 0.61) (#60880)

Backport PR #60847: TST/CI: skipif numba tests on Ubuntu ARM for numba 0.61
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
numba numba-accelerated operations Testing pandas testing functions or related to the test suite
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants